Modern computing and display technologies have facilitated the development of “mixed reality” (MR) systems for so called “virtual reality” (VR) or “augmented reality” (AR) experiences, wherein digitally reproduced images or portions thereof are presented to a user in a manner wherein they seem to be, or may be perceived as, real. A VR scenario typically involves presentation of digital or virtual image information without transparency to actual real-world visual input. An AR scenario typically involves presentation of digital or virtual image information as an augmentation to visualization of the real world around the user (i.e., transparency to real-world visual input). Accordingly, AR scenarios involve presentation of digital or virtual image information with transparency to the real-world visual input.
MR systems typically generate and display color data, which increases the realism of MR scenarios. Many of these MR systems display color data by sequentially projecting sub-images in different (e.g., primary) colors or “fields” (e.g., Red, Green, and Blue) corresponding to a color image in rapid succession. Projecting color sub-images at sufficiently high rates (e.g., 60 Hz, 120 Hz, etc.) may deliver a smooth color MR scenarios in a user's mind.
Various optical systems generate images, including color images, at various depths for displaying MR (VR and AR) scenarios. Some such optical systems are described in U.S. Utility patent application Ser. No. 14/555,585 filed on Nov. 27, 2014, the contents of which are hereby expressly and fully incorporated by reference in their entirety, as though set forth in full.
MR systems typically employ wearable display devices (e.g., head-worn displays, helmet-mounted displays, or smart glasses) that are at least loosely coupled to a user's head, and thus move when the user's head moves. If the user's head motions are detected by the display device, the data being displayed can be updated to take the change in head pose (i.e., the orientation and/or location of user's head) into account.
As an example, if a user wearing a head-worn display device views a virtual representation of a virtual object on the display and walks around an area where the virtual object appears, the virtual object can be rendered for each viewpoint, giving the user the perception that they are walking around an object that occupies real space. If the head-worn display device is used to present multiple virtual objects, measurements of head pose can be used to render the scene to match the user's dynamically changing head pose and provide an increased sense of immersion. However, there is an inevitable lag between rendering a scene and displaying/projecting the rendered scene.
Head-worn display devices that enable AR provide concurrent viewing of both real and virtual objects. With an “optical see-through” display, a user can see through transparent (or semi-transparent) elements in a display system to view directly the light from real objects in an environment. The transparent element, often referred to as a “combiner,” superimposes light from the display over the user's view of the real world, where light from by the display projects an image of virtual content over the see-through view of the real objects in the environment. A camera may be mounted onto the head-worn display device to capture images or videos of the scene being viewed by the user.
Current optical systems, such as those in MR systems, optically render virtual content. Content is “virtual” in that if does not correspond to real physical objects located in respective positions in space. Instead, virtual content only exist in the brains (e.g., the optical centers) of a user of the head-worn display device when stimulated by light beams directed to the eyes of the user.
MR systems attempt to present color, photo-realistic, immersive MR scenarios. However, lag time between generation of virtual and display of the generated virtual content combined with head movement during the lag time can result in visual artifacts (e.g., glitches) in MR scenarios. This problem is exacerbated by rapid head movement during the lag time and with color content generated by sequentially projecting colors or fields (i.e., sequential displays such as LCOS).
In order to address this issue, some optical systems may include a warping software/system that receives source color virtual content from a source. The warping system then “warps” (i.e., transforms the frame of reference of) the received source color virtual content for display in a frame of reference of the display or output system/viewer (the “display or output frame of reference”). Warping or transforming changes the frame of reference from which color virtual content is presented. This approach takes the originally rendered color content, and shifts the way that the color content is presented to attempt to display the color content from a different perspective.
Some warping software/systems warp the source virtual content in two processing passes. Warping systems warp all of the source subparts forming a 3-D scenario in the source virtual content in a first pass. The warping systems also perform depth testing in this first pass to generate depth data, but the depth testing is performed in the source frame of reference. The warping systems store all the warped subparts resulting from the transformation of the source subparts forming the 3-D scenario and their relative depths in the source frame of reference in that first pass (e.g., in a list).
During warping, two or more different subparts of a 3-D scenario may warp/project into (i.e., be assigned to) the same pixel of a final display image. These subparts are “conflicting,” and the warping system must resolve the conflict to generate a realistic 2-D display image.
After the first pass, some of the warped subparts may be conflicting relative to pixels of the final 2-D display image. The warping systems then perform a second pass through the intermediate warping data stored in the first pass to analyze the depth test data of conflicting warped subparts to identify the warped subparts closest to the viewing location in the output frame of reference. The conflicting warped subpart closest to the viewing location in the output frame of reference is used to generate a final 2-D display image. The remaining conflicting warped subparts are discarded.
Some warping software/systems warp color source virtual content using the same X, Y location in the output frame of reference for all colors/fields in the same color image. However, using one X, Y location in the output frame of reference to warp all (e.g., three) colors/fields ignores the close, but nonetheless different, times at which the different color sub-images are projected. This results in visual artifacts/anomalies/glitches that can detract from the immersiveness and realism of MR systems.